Skip to content

fix(pr-pipeline): skip a lane's key leg while its key is out of credit - #12617

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/lane-key-credit-gate
Nov 3, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
fix/lane-key-credit-gate

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #12526

Summary

The API key the lanes try first has no credit, so every key-credit leg fails in one turn (Credit balance is too low) and the subscription leg does the work. Baseline from #12526: 99 of 218 model legs (45%) on 2026-11-02 18:08-22:05Z, including check-intent 61/61, review-claude-review 18/18 and review-claude-security-review 18/18. ci-workflows' hosted review lanes already skip the key after a credit stop (check-key-credit; on #12508 security-review-key-credit passed, the key job skipped and security-review-subscription reviewed). This repository's own pr-verify.yml and pr-review-main.yml had no such gate. They now reuse the same action.

Fix

  • pr-verify.yml: new check-intent-credit job (actions: read only, no secret, no checkout) runs check-key-credit for check-intent. While the key's credit stop is inside the window, check-intent is skipped and check-intent-subscription judges with fallback reason credit-stop. check-intent records its own credit stop in a step named for the run; the probe carries the origin run forward, so the window counts from the stop. Any probe failure runs the key (!cancelled()).
  • pr-review-main.yml: the same through review-handoffs-credit for the review matrix (both lanes share REVIEWS_CLAUDE_API_KEY); review-handoffs routes a gate-skipped key to the subscription leg on the trigger's main commit. The probe runs on every run so the record is not lost from the gate's three-run read.
  • A gate-skipped key leg leaves a lane-outcome record (outcome skip, class credit-stop, handoff true) under the same lane id, so leg counts compare before and after.
  • One window, externalized: vars.KEY_CREDIT_WINDOW_HOURS, default 1 hour (an at-most-hourly re-probe, per fix(lanes): the CI API key is out of credit, so 45% of model legs fail instantly while reporting green #12526). The variable is not created.
  • Action pinned at ci-workflows 2476d5d, the pin the other ci-workflows actions here share (pr-verify/workflow.test.mjs:694-703).
  • adjudicate-feedback is unchanged: it runs on the key only, with no subscription leg.
  • New .github/scripts/lib/gate-expression.mjs: a small evaluator for job-level if: expressions, shared by both workflow tests.

Done-when item 1 (credit-stop recorded in telemetry) was already met for legs that run; item 2 is met in the workflow text and is measured after merge against the baseline above. Item 3 (#6958, key rotation) is the owner's.

Hardening residuals (no token, account or billing risk; s10):

  • a. gate.cjs settles a clean key run only by a step named Claude review (MODEL_STEP, gate.cjs:35); these workflows' model steps are Judge the pull request and Review chunk N, so a clean run does not cut a window short. Cost: at most one extra 1-hour skip window and up to 3 job-list reads per probe. Fix later as a model-step input on the action.
  • b. The gate lists completed runs only, so runs started while a credit-stop run is still in flight still try the key.
  • c. A lane turned off by its lane switch is not routed to the subscription leg while the gate skips.

Verification

  • node --test over pr-verify/workflow, pr-verify/probe/parity, pr-review-main/workflow, maintenance-reconcile-lost-runs/reconcile, log-api-budget, resolve-config/callers and pr-merge/workflow tests: 656/656 pass.
  • scripts/check-affected.sh --base origin/main --no-fetch: 10 passed, 5 skipped.
  • zizmor 1.30.0 offline: no findings; actionlint 1.7.12: only the existing cache-mode key notice.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcupojVgxFqBnrah9DwFLE

kyle-sexton and others added 4 commits November 2, 2026 21:28
pr-verify and pr-review-main tried the API key first on every run, and
with the key out of credit each key leg ended in one turn on "Credit
balance is too low" before the subscription leg did the work (#12526).

Each workflow now runs ci-workflows' check-key-credit in a probe job
before its key leg (check-intent-credit, review-handoffs-credit): actions
read only, no secret, no checkout. A key-leg credit stop recorded inside
the window (vars.KEY_CREDIT_WINDOW_HOURS, 1 hour when unset) skips the key
job, and the subscription leg runs as for a credit-stop handoff. The
probe's record step carries the origin run forward, so the key is tried
again once the window ends and a topped-up or rotated key is picked up
with no edit. Any probe failure leaves skip-key empty and the key runs:
each key job's `if` starts with !cancelled().

- pr-verify: check-intent records the stop after a judge step's
  credit-stop handoff; check-intent-subscription and check-intent-fallback
  spell out the success their implicit success() gave, and the
  subscription's fallback-reason names credit-stop for a gated key.
- pr-review-main: both lanes share the reviews key, so one gate skips the
  review job; review-handoffs records a key leg's credit stop, hands every
  enabled lane on as credit-stop at the trigger job's main commit when the
  gate skipped the key, and the record job runs when the subscription leg
  did.
- A gated key leg leaves a lane-outcome record (outcome skip, class
  credit-stop, handoff true) from the probe, so leg counts compare before
  and after.

The tests evaluate the new job gates through a shared expression
evaluator and pin the record step names to the gate's RECORD_PATTERN.

Refs: #12526

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VcupojVgxFqBnrah9DwFLE
check-key-credit reads only the newest three completed, unskipped runs.
Most pr-review-main runs end success with no lane source to review, so a
gate that ran only on lane-source runs left those runs without the carried
record, and the origin fell out of the gate's reads before the window ended.

review-handoffs-credit now runs on every run of this attempt's trigger job.
Only a run whose trigger job starts the key legs reports them skipped or
lets review-handoffs hand the lanes on, so a run with no lane source still
routes nothing.

Refs: #12526

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VcupojVgxFqBnrah9DwFLE
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 3, 2026 03:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T03:56:02.371600Z 17afcda Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 17afcda6d6b3b33624f5d2bc30f7089f329da3b5: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

critical-findings: 0

I read every hunk of all 6 changed files. The two workflow files are trusted paths, so I read their whole change and what it gates.

What I checked

  • New credit-gate jobs (check-intent-credit, review-handoffs-credit):
    • Each job's only permission is actions: read. It references no secret, does no checkout and starts no model.
    • The gate action is pinned to a full SHA in the owner's own ci-workflows repository: check-key-credit@2476d5dd…, at pr-verify.yml#L871.
    • origin-run and github.run_id appear only in step names. Every run: body is a fixed echo, so no expression reaches a shell.
  • What skip-key changes. It only picks which credential leg judges. Even if it were set wrongly, the work just moves to the subscription leg, and that leg still runs all of its own gates:
    • In pr-verify, check-intent-subscription still runs the kill switch, resolve-pr.mjs and check-trusted-trigger against the trusted-actor list (pr-verify.yml#L2089-L2124).
    • When the key job is skipped, CARRIED is empty, so resolve-pr resolves the pull request itself; the comment at L2100 says it takes the carried reads only when they hold.
    • check-intent-subscription keeps its trigger gate (same-repository pull_request run on main). The fallback job now requires needs.check-intent-subscription.result == 'success' explicitly.
  • Changed if: conditions. The key jobs' new !cancelled() plus an explicit needs.trigger.result == 'success' keep the same trigger, fork and ref conditions. A failed probe runs the key leg as before, so nothing new gets through.
  • New route in pr-review-main. review-handoffs keeps permissions: {}. The main-sha fallback comes from the trigger job's checkout of the default branch. The route's jq still accepts only the reasons credit-stop auth no-key and only a base matching ^[0-9a-f]{40}$ (pr-review-main.yml#L1264-L1271). review-subscription keeps its own trigger gate and read-only permissions.
  • record job. It now also runs when the key leg was skipped but the subscription leg ran. It still records only what write-verdict.mjs checked from this run's own artifacts.
  • Trust controls. Nothing widens:
    • who passes the trusted-trigger check;
    • lane token scope;
    • the kill switch (every model leg still checks it);
    • merge holds (no hold file was touched).
  • Instruction surfaces and plugin manifests. None changed.
  • Test files. gate-expression.mjs and the three test files only run in tests and have no security surface.

What it spends: the subscription leg now does the work while the key is out of credit, which spends subscription usage. Each run also adds one short probe job on a standard ubuntu-24.04 runner. No paid plan or billing setting changes.

The pull request description contains no embedded instructions to this lane.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

claude-review reviewed 17afcda6d6b3b33624f5d2bc30f7089f329da3b5: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

I read every hunk of all 6 changed files in .claude-lane/pr.diff. I also read the job graphs of pr-verify.yml and pr-review-main.yml at head to check every job downstream of a key leg the gate skips.

What changes. A new job in each workflow (check-intent-credit and review-handoffs-credit) runs ci-workflows check-key-credit. While the key's last credit stop is inside the window, that job skips the key job and hands the work to the subscription leg as a credit-stop. A shared test helper, .github/scripts/lib/gate-expression.mjs, evaluates job if: expressions for the tests.

What I checked, with nothing to report:

  • Skipped key job in pr-verify.yml. Every job downstream now tolerates a skipped check-intent:
    • check-intent-subscription adds !cancelled() and checks needs.check-intent.result == 'success' explicitly.
    • check-intent-fallback adds !cancelled() and checks the subscription job's success explicitly.
    • wake-pr-merge already used !cancelled().
    • An empty needs.check-intent.outputs.reads makes resolve-pr.mjs do its own reads ("taken only when they hold").
  • Skipped key job in pr-review-main.yml.
    • review checks needs.trigger.result == 'success' in place of the implicit success() it had before.
    • review-subscription already used !cancelled() together with needs.review-handoffs.result == 'success'.
    • The new condition on record runs it only when a key or subscription leg could have reviewed.
    • review-handoffs hands lanes on only when this attempt's trigger job would have started the key leg. The trigger's files step runs only when at least one lane is enabled, so its review == 'true' output means at least one lane is on. That keeps the route's matrix from coming out empty.
    • CODE_BASE/SECURITY_BASE fall back to the new main-sha output, which a step writes right after the trigger job's checkout.
  • Whether a probe failure blocks review. It does not. The probe step has continue-on-error, and the key job's if starts with !cancelled(), so a failed or skipped probe leaves skip-key empty and the key runs as before.
  • Where the credit records are written. Each key leg records its own stop only after a credit stop (always() / !cancelled() plus the handoff-reason checks). A lane the gate handed on writes no new record in review-handoffs, so the window still counts from the original stop.
  • gate-expression.mjs. It gives && higher precedence than || and consumes every token, and it asserts that the whole gate was tokenized and parsed. It matches GitHub's semantics for every shape these workflows use.

Not verified (no shell in this lane). The test, zizmor and actionlint results in the PR body are the author's claims; I did not run them.

Unverified observation, not counted as a finding. The PR body says the probe runs on every run so the record stays inside the gate's three-run read. In pr-review-main.yml, review-handoffs-credit has an if that still requires workflow_run.conclusion == 'success' and a same-repository pull_request run. If check-key-credit counts runs without the job toward its read, a few such runs in a row could drop the record. The cost would be one extra key leg that fails and records again. Whether this happens depends on gate.cjs in ci-workflows, which I could not read.

Criteria sources. The PR only references #12526 and closes no issue, so there are no acceptance criteria to check. No .claude/rules/ file's paths: matches a changed file. No plugin manifest changed. There are no prior findings, so the round cap does not apply. I found no embedded instructions in the PR context.

Spend (not a hold). The PR adds one small job per run in each workflow, with no model and no secret, on ubuntu-24.04. It cuts key-leg API calls that would fail on no credit; the work moves to subscription legs that already run today.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 0fa4041 Nov 3, 2026
74 checks passed
@kyle-sexton
kyle-sexton deleted the fix/lane-key-credit-gate branch November 3, 2026 04:17
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File Killed Survived Timeout No coverage Invalid Not run
.github/scripts/lib/gate-expression.mjs 5 2 0 0 0 0

Surviving mutants

Location Operator Disposition Cause
.github/scripts/lib/gate-expression.mjs:9 SBR productive input-gap
.github/scripts/lib/gate-expression.mjs:43 SBR productive input-gap

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant